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SIMMRY 


The  purpose  of  this  study  was  to  assess  the  possibility  of  obtaining  an  overall  ranking  of  12 
AFHRL  Advanced  Technology  Development  Projects  sufficiently  reliable  to  be  used  as  a  reference 
document  for  fund  allocations,  manpower  allocations,  and  program  cuts.  A  second  objective  was  to 
determine  what  criteria  Judges  used  In  ranking  the  projects  on  overall  merit. 

Seven  Judges  ranked  the  projects  twice  on  six  factors.  The  Interjudge  agreement  In  ranking 
the  projects  on  an  Overall  Assessment  (OA)  factor  for  Session  1  was  _r  *  .47  (p^  <  .001).  For 
Session  2,  _r  ■  .60  <  .001).  Eliminating  a  divergent  Judge  Increased  the  correlations  to  .57 

and  .69,  respectively.  Interjudge  agreement  Increased  across  sessions  (£  <  .01),  presumably  from 
the  effects  of  feedback  and  changing  one  of  the  rating  factors  In  Session  2. 

The  OA  rankings  for  Session  1  did  not  differ  significantly  from  those  of  Session  2  (£  *  .50). 
The  rank-order  correlation  between  the  rankings  for  Sessions  1  and  2  was  .97  <  .001).  Ty/o 

additional  Judges  who  ranked  the  projects  on  only  the  OA  factor  gave  rankings  similar  to  those  of 
the  group.  In  half  the  time. 

The  factors  most  highly  correlated  with  the  OA  factor  were  Customer  Interest  and  Payoff  to 
the  Air  Force.  Cost,  Schedule,  and  Technical  Feasibility  mey  not  have  been  significantly 
correlated  with  OA  because  the  Project  Descriptions  used  by  the  Judges  did  not  adequately 
describe  the  projects  In  terms  of  the  factors  on  which  the  projects  were  rated. 

Recomnendatlons  Include  having  project  scientists  prepare  written  descriptions  of  projects  In 
terms  of  the  factors  on  which  the  projects  will  be  ranked;  using  different  rating  factors  or 
procedures  for  new  projects  as  opposed  to  older  projects;  ranking  projects  on  an  OA  factor  only; 
and  keeping  records  of  project  rankings  for  several  years  so  that  ranking  validities  can  be 
assessed.  Interjudge  rankings  were  sufficiently  reliable  to  suggest  that  project  rankings  based 
on  expected  Program  Payoff  to  the  Air  Force  can  be  used  as  a  reference  In  allocating  research  and 
development  resources. 
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PRIORITIZING  RESEARCH  AND  DEVELOPMENT  PROJECTS  OF  THE 
AIR  FORCE  HUMAN  RESOURCES  LABORATORY 


I.  INTRODUCTION 

Research  and  Development  (RSO)  resources  are  often  distributed  to  organizational  units  In  the 
same  proportion  each  year,  based  on  historical  precedent.  Money  and  manpower  are  distributed 
proportionately,  and  little  consideration  Is  given  to  basing  the  allocation  on  new  program 
priorities.  While  distributing  resources  proportionately  Is  expedient.  It  Ignores  the  fact  that 
priorities  change  over  time. 

An  alternative  method  of  distributing  resources  Is  to  base  the  allocation  on  program  merit  as 
assessed  by  a  group  of  experts.  In  this  study*  seven  Judges  ranked  12  AFHRL  Advanced  Technology 
Development  Projects  on  six  factors  during  two  ranking  sessions.  The  projects  covered  the  areas 
of  Personnel  Selection  and  Classification,  Technical  Training,  Flying  Training,  and  Logistics. 
The  objective  of  the  study  was  to  determine  If  It  was  possible  to  obtain  an  overall  ranking  of 
the  12  projects  which  was  sufficiently  reliable  to  be  used  as  a  reference  document  for  fund 
allocations,  manpower  allocations,  and  program  cuts.  A  secondary  objective  was  to  determine  the 
basis  on  which  the  Judges  made  their  rankings.  No  attempt  was  made  to  establish  the  validity  of 
the  rankings. 

Although  AFHRL  had  previously  used  project  rankings  In  distributing  resources,  the 
reliability  and  underlying  rationale  for  the  rankings  was  unknown.  In  this  study,  measures  of 
rater  reliability  were  obtained,  and  the  rationale  underlying  the  Judges'  rankings  was  examined. 
The  rating  factors  employed  were  developed  specifically  for  ranking  Advanced  Development  Projects 
(6.3A),  and  were  derived  from  recommendations  from  the  judges  used  In  this  stuty  and  from  other 
studies  which  assessed  assigning  priorities  to  RAD  projects  (Doherty  A  Seymour,  1983). 


II.  METHOD 


Subjects  and  Design 

Seven  males  volunteered  to  participate  as  Judges.  Three  were  from  Headquarters,  Human 
Systems  Division  (HQ  HSD),  and  four  were  from  AFHRL.  Each  Judge  held  a  position  of 
responsibility  at  the  headquarters  level,  either  as  a  planner,  progranmer,  or  plans  analyst.  Six 
were  civilians,  grades  GM/6S-13  and  -14;  one  was  an  Air  Force  major.  Four  had  doctorate  degrees 
and  three  had  masters  degrees.  All  were  familiar  with  the  RAD  projects  through  readings, 
briefings,  and  program  management  activities.  Each  Judge  ranked  12  RAD  projects  twice  on  six 
factors.  The  second  ranking  followed  a  feedback  session  held  approximately  2  months  after  the 
first  ranking  session. 

Materials 


Each  Judge  received  a  packet  consisting  of  Instructions,  Project  Descriptions,  Ranking  Forms, 
and  a  Questionnaire.  The  Project  Descriptions,  formally  called  Descriptive  Summaries,  covered 
all  AFHRL  Advanced  Development  Projects  In  program  elements  63106F,  63227F,  and  63751F. 

Descriptive  Summaries  are  documents  prepared  by  AFHRL,  HQ  HSD,  and  HQ  Air  Force  Systems  Command 
(HQ  AFSC),  for  use  by  Air  Force  management  and  the  US  Congress  In  allocating  funds  to  RAD 
projects.  All  judges  In  the  study  were  very  familiar  with  the  Descriptive  Summaries  and  some 
even  had  a  role  In  preparing  them. 

Each  Judge  received  six  Ranking  Forms  stapled  together.  The  order  of  the  pages  was 
counterbalanced  across  Judges.  Each  form  listed  12  projects  and  had  one  factor  definition  at  the 
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top  of  the  page.  Project  order  was  scrambled  so  that  each  judge  had  a  different  project  ordering 
on  each  page.  Five  rating  factors  were  scrambled  such  that  each  Judge  had  a  different  factor 
order  across  pages.  The  sixth  factor.  Overall  Assessment  (OA),  was  always  the  last  factor  on 
which  the  projects  were  ranked. 


Procedure 

A  preliminary  meeting  was  held  with  all  seven  Judges.  The  purpose  of  the  study  was 

explained,  and  each  Judge  was  given  a  list  of  sample  factor  definitions.  They  were  then  asked  to 
recomiend  10  factors  on  which  to  rate  the  projects.  The  factors  selected  were  very  similar  to 
those  recommended  by  the  Judges,  and  were  as  follows: 

1.  FEASIBILITY  OF  SCHEDULE  (FS);  The  PAD  schedule  Is  likely  to  be  met  without  the  use  of 
additional  unplanned  resources. 

2.  TECHNICAL  FEASIBILITY  OF  PLANS  (TF):  The  RAD  plans  are  technically  feasible,  well 

defined,  easy  to  understand,  and  seem  to  call  for  a  logical  sequence  of  events. 

3.  CUSTOMER  INTEREST  AND  INVOLVEMENT  (Cl):  The  customer  Is  Interested  In,  and  actively 

Involved  In,  the  RAD  planning  and  programming. 

4.  FEASIBILITY  OF  PROOUCT  TRANSITION  (FT):  Product  transition  plans  are  realistic, 
practical,  affordable,  and  likely  to  occur  as  planned. 

5.  REASONABLENESS  OF  COST  (RC):  RAD  cost  Is  reasonable  and  affordable  considering  the 

expected  payoff  to  the  Air  Force. 

6.  OVERALL  ASSESSMENT  lOA):  All  factors  considered,  rank  the  projects  from  highest  (1)  to 
lowest  (12)  priority. 

Session  1:  Two  weeks  after  the  preliminary  meeting,  all  Judges  were  mailed  the  Instructions, 
the  Project  Descriptions,  the  Rating  Forms,  and  the  Questionnaire.  They  were  Instructed  to  read 
the  Project  Descriptions  and  then  rank  the  projects  on  each  of  the  six  factors  selected  by  the 
authors.  Rankings  were  to  be  based  on  Project  Descriptions  plus  any  other  personal  Information 
the  Judges  had  from  briefings  and  other  readings.  Judges  were  asked  to  return  all  forms  within  7 
days  of  receipt. 

Session  2:  Approximately  2  months  after  the  due  date,  five  of  the  Judges  ranked  the  projects 
again,  after  attending  a  90-m1nute  meeting  during  which  they  were  given  the  results  of  their 
previous  rankings.  Each  Judge  was  shown  every  other  Judge’s  OA  ranking  for  the  12  projects,  plus 
the  average  of  those  rankings  for  each  project.  Discussion  centered  on  the  rationale  for  the 
rankings  assigned  and  the  definition  of  the  factors. 

As  a  result  of  the  Judges'  recommendations,  one  rating  factor  was  added  and  one  was  deleted 
for  the  second  ranking  session.  "Payoff  to  the  Air  Force"  (PO)  was  added,  and  "Feasibility  of 
Schedule"  (FS)  was  deleted.  "Payoff  to  the  Air  Force"  was  defined  as  "Potential  of  the  RAD 
product  to  Increase  the  efficiency  and  effectiveness  of  Air  Force  operations."  The  factors  were 
defined  precisely,  so  that  each  Judge  would  have  the  same  framework  for  ranking  during  Session  2. 

The  two  Judges  who  were  unable  to  attend  the  group  session  attended  Individual  30-m1nute 
feedback  sessions  with  the  primary  author  a  few  days  following  the  group  meeting.  Thus,  all 
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judges  attended  a  feedback  session  prior  to  Session  2.  This  procedure  could  be  viewed  as  a 
Modified  Delphi  technique  (Oalkey,  1969).  No  attempt  was  made  to  achieve  a  group  ranking 
consensus. 


III.  RESULTS 

Analysis  of  Overall  Assessment  (OA)  Ranks:  Table  1  shows  the  DA  ranks  Judges  assigned  to  the 
projects  during  each  ranking  session.  The  Kendall  Coefficient  of  Concordance  (Siegel,  1956)  gave 
an  average  Interjudge  correlation  of  _r  ■  .47,  X^di,  «  7)  .  41.81,  £  <  .001  for  Session  1, 
and  _r  ■  .60,  x^(11,  “  7)  *  50.25,  £  <  .001  for  Session  2.  Agreement  among  judges  during  both 

sessions  was  moderately  high. 

A  comparison  of  the  average  pairwise  Pearson  correlations  among  the  OA  rankings  of  the  seven 
judges  across  sessions  (Table  2)  shows  higher  Intercor relations  during  Session  2  than  Session  1. 
Applying  the  Sign  test,  the  probability  of  this  difference  being  due  to  chance  is  £  »  .008 
(one-tailed).  The  expectation  was  that  there  would  be  greater  Interjudge  agreement  following 
feedback,  and  there  was.  The  change  In  one  rating  factor  from  Session  1  to  Session  2  may  also 
have  contributed  to  the  Increase  In  Interjudge  agreement. 

An  analysis  was  also  performed  to  see  If  there  were  any  divergent  raters.  Judge  3  was 
divergent  because  none  of  his  OA  rankings  was  significantly  correlated  (o  »  .05)  with  any  other 
judge's  rankings  In  either  Session  1  or  Session  2.  Dropping  Judge  3  from  the  analysis  raised  the 

Session  1  average  Intercorrelatlon  among  judges  from  .47  to  .57,  and  the  Session  2  average 

Intercorrelatlon  from  .60  to  .69. 

The  correlation  between  the  average  Session  1  and  the  average  Session  2  OA  ranks  (Table  1, 
last  two  columns)  was  £  (10)  *  .97,  £■  13.08,  £  <  .001,  Indicating  high  similarity  between  ranks 
across  sessions,  despite  the  change  of  one  rating  factor  and  the  group  discussion.  As  can  be 
seen  In  the  last  column  of  Table  1,  seven  projects  had  the  same  average  rank  In  both  sessions, 
and  all  but  one  of  the  remaining  five  projects  changed  by  only  one  rank.  Project  1  changed  by 
two  ranks;  It  was  ranked  1  in  Session  1,  and  3  In  Session  2.  The  correlation  between  all  Session 
1  ranks  and  all  Session  2  ranks  (84  pairs)  was  £  (82)  »  .92,  Jt  »  7.32,  £  <  .001,  further 
Indicating  the  similarity  of  the  rankings  between  sessions.  A  Sign  test  applied  to  the  change  In 
average  OA  ranks  (second  from  the  last  column  of  Table  1),  gives  £  >  .50,  Indicating  no 

significant  difference  In  average  OA  ranks  across  sessions. 

()uest1onna1re  Analysis;  Principal  findings  from  analysis  of  the  seven  completed 
(Juestlonnalres  were  as  follows: 

1.  Judges  reported  that  the  Project  Descriptions  (Descriptive  Sunmarles)  contained 

Insufficient  Information  regarding  funding,  product  Implementation,  and  technical  aspects  of  the 
projects  to  be  useful  In  ranking  the  projects.  On  a  6-po1nt  scale  (1  ■  Adequate,  6  » 
Inadequate),  they  rated  Project  Descriptions  as  Inadequate  (Mean  (^)  »  5.7)  for  use  In  performing 
the  ranking  task.  As  a  consequence,  they  based  their  ratings  primarily  on  personal  familiarity 
with  the  projects  (j1  ■  1.3).  Some  judges  said  the  Descriptive  Summaries  were  also  Inadequate  for 
the  purpose  for  which  they  were  originally  Intended  (l.e.,  program  justification). 

2.  The  rating  factors  employed  during  Session  1  were  judged  to  be  midway  (M,  *  3.0)  between 
Very  Appropriate  (1)  and  Very  Inappropriate  (6).  One  factor  was  deleted  and  one  was  added,  as 
described  earlier. 

3.  The  judges  took  an  average  of  2  hours  to  complete  the  rankings  and  (Xiestlonnalre.  They 
found  the  ranking  task  to  be  about  midway  (J4  ■  3.1)  between  Very  Difficult  (1)  and  Very  Easy  (6), 
and  they  were  midway  (M  •  4.1)  between  Confident  (1)  and  Mot  At  All  Confident  (6)  about  their 
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rankings.  They  reported  they  were  Quite  Familiar  (1)  with  the  R40  projects  *  2.0).  Four  of 
the  seven  judges  believed  that  the  rankings  from  the  study  should  be  used  In  making  fund 
distributions,  funding  cuts,  and  manpower  allocations.  Five  projects  were  recommended  for 
termination  and  six  were  recoamended  for  Increased  support.  For  two  of  these  projects,  the 
recoamendatlons  overlapped;  that  Is,  some  Judges  recoamended  termination  and  others  recoamended 
Increased  support.  Three  projects  received  no  specific  recommendation. 

Table  2.  Average  Pairwise  Intercorrelatlons  on  OA  Factor 


Judges 

1 

2 

3 

4 

5 

6 
7 

8« 

9« 


Session  1 
.45 
.49 
.21 
.52 
.55 
.56 
.49 
.53 
.38 


Session  2 
.60 
.54 
.35 
.73 
.71 
.66 
.57 
.52 
.35 


^Evaluations  completed  following  the  study,  with  two 
Judges  ranking  on  the  OA  factor  only  (see  DISCUSSION). 

Analysis  of  the  Relationships  Among  Factors;  Tables  3  and  4  show  the  average  rankings  of  the 
seven  Judges  for  all  projects  and  factors  for  both  sessions.  Separate  multiple  regression 
analyses  were  performed  on  these  rankings  to  determine  the  relationship  between  the  OA  rankings 
and  the  rankings  on  the  five  other  factors.  For  Session  1,  ^  (11)  ■  .98,  F  «  24.74,  <  .001. 

For  Session  2,  It  was  also  .98  (11)  ■  .98,  F  ■  30.63,  £  <  .001].  A  stepwise  regression 

analysis  selected  Customer  Interest  (Cl)  as  the  factor  most  highly  correlated  with  OA  for  Session 
1  [£  (11)  ■  .91,  F  ■  46.83,  £  <  .001],  and  Payoff  to  the  Air  Force  (PO)  and  Feasibility  of 
Product  Transition  (FT)  as  the  most  highly  correlated  factors  for  Session  2  (11)  ■  .97,  F  « 

44.23,  2.  <  .001].  Without  entering  the  FT  factor,  the  correlation  between  PO  and  OA  was  _r  (11)  « 
.96,  F  ■  85.92,  £  <  .001.  Thus,  Cl  and  PO  were  primary  predictors  of  OA,  perhaps  because  the 
Judges  had  Inadequate  Information  about  cost,  schedule,  and  the  other  factors,  so  that  It  was 
necessary  to  rank  on  more  global  factors. 

Table  3.  Mean  Ranks,  Session  1 


Projects 

1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

n 

12 


Factors 

FT 


Table  4.  Mean  Ranks,  Session  2 


Project'. 

Factors 

RC 

FI 

FT 

Cl 

PO 

OA 

1 

4.4 

3.3 

6.6 

6.1 

7.6 

6.6 

2 

12.0 

7.0 

8.6 

11.3 

11.9 

11.9 

3 

6.3 

7.3 

2.1 

5.6 

4.7 

5.6 

4 

5.3 

6.7 

5.6 

6.4 

2.1 

2.4 

5 

6.7 

8.7 

9.6 

7.7 

6.3 

8.0 

6 

2.9 

3.1 

2.9 

1.7 

3.7 

2.9 

7 

5.1 

5.1 

6.7 

3.9 

4.7 

5.4 

8 

7.1 

7.3 

7.0 

5.1 

7.1 

7.6 

9 

4.3 

4.6 

5.0 

2.4 

3.7 

3.0 

10 

8.9 

10.0 

9.0 

10.1 

10.6 

10.1 

11 

8.3 

8.4 

6.9 

10.0 

9.3 

8.4 

12 

6.4 

6.4 

6.6 

7.7 

6.4 

6.1 

IV.  DISCUSSION 

The  principal  finding  Is  that  there  was  sufficient  Interjudge  agreement  to  conclude  that 
project  rankings  can  be  used  as  a  reliable  reference  In  allocating  RAO  resources.  The  fact  that 
the  Project  Descriptions  contained  Insufficient  detail,  and  judges  reverted  to  prior  knowledge  In 
making  their  evaluations,  Indicates  that  Project  Descriptions  should  be  tailor-made  when  used  as 
a  reference  In  ranking  projects.  They  should  be  tailored  to  Include  the  merits  of  the  projects 
In  terms  of  the  factors  on  which  the  projects  are  to  be  evaluated.  Describing  the  projects  In 
terms  of  these  factors  may  eliminate  some  of  the  judging  bias  which  can  result  from  prior 
knowledge.  Seme  prior  knowledge  In  this  study  seemed  to  be  based  on  hearsay  rather  than  personal 
knowledge  of  the  technical  merits  of  the  projects.  Project  Descriptions  should  be  prepared  In  a 
cotmnon  format  by  project  scientists.  Using  a  common  format  may  eliminate  some  of  the  bias 
Introduced  by  different  writing  styles. 

Interjudge  agreement  on  the  OA  rankings  Increased  from  Session  1  to  Session  2,  but  there  was 
not  a  significant  difference  between  average  OA  ranks  across  sessions.  However,  the  project 
ranked  1  In  Session  1  was  ranked  3  In  Session  2,  which  Is  an  Important  change.  Changes  like 
this,  plus  the  fact  that  Interjudge  reliability  increased  following  the  feedback  session. 
Indicate  that  feedback  had  an  Important  effect.  The  total  effect  of  the  feedback  Is 
undetermined.  Dalkey  (1969)  has  found  that  face-to-face  discussions,  like  the  one  In  this  study, 
tend  to  make  group  estimates  less  accurate  than  those  following  anonymous  controlled  feedback. 
Our  Impression  was  that  the  face-to-face  discussion  was  beneficial,  because  It  allowed  judges  to 
share  Infonnatlon  and  possibly  make  more  knowledgeable  judgments.  Dalkey' s  task  was  very 
different  from  the  one  performed  In  this  study.  His  subjects  estimated  known  facts  of  the  sort 
found  In  an  almanac,  such  as  "How  many  telephones  were  In  use  In  Africa  In  1965?*  Very  Hkely 
the  effects  of  feedback  are  related  to  the  nature  of  the  subjects'  task. 

Based  on  comments  made  during  the  feedback  session.  It  may  be  wise  to  use  different  ranking 
factors  for  new  versus  older  projects.  It  Is,  for  example,  more  difficult  to  make  judgments 
about  cost,  schedule,  and  technical  risk  for  new  projects  than  for  projects  which  have  been 
underway  Tor  2  or  3  years.  In  fact.  It  may  be  appropriate  to  use  an  entirely  different  resource 
allocation  procedure  for  new  projects.  Small  amounts  of  seed  money  could  be  provided  for  several 
new  projects,  rather  than  large  amounts  of  money  for  a  few  highly  ranked  projects.  This  risk 
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averaging  technique  may  be  the  best  course  of  action  In  cases  of  high  uncertainty.  As  the 
projects  mature,  a  ranking  procedure  could  be  adopted. 


Similar  OA  rankings  might  have  been  obtained  without  having  the  judges  first  rank  the 
projects  or.  i,ne  other  five  factors.  This  notion  was  tested  by  having  two  additional  judges  rank 
the  projects  only  on  OA.  They  were  told  to  rank  the  projects  on  Overall  Payoff  to  the  Air  Force, 
considering  Reasonableness  of  Cost,  Feasibility  of  Schedule,  Customer  Interest  and  Involvement, 
Technical  Feasibility  of  Plans,  and  Feasibility  of  Product  Transition.  They  ranked  the  projects 
once,  and  then  again  after  seeing  the  OA  ranks  for  all  judges  in  Session  1;  there  was  no 
feedback  session.  One  judge's  ranks  correlated  £  (10)  »  .52,  ^  *  1.91,  >  .05,  (two-tall)  with 

average  Session  1  ranks,  and  (10)  •  .50,  ^  »  1.84,  p,  >  *05  with  average  Session  2  ranks.  The 
second  judge's  ranks  correlated^  (10)  »  .72,  ^  •  3.28,  £  <  .01  for  Session  1,  and  r  (10)  «  .66, 
^  ■  2.76,  £  <  .05  for  Session  2.  There  was  little  or  no  difference  between  their  first  and 
second  set  of  ranks.  One  judge  made  two  rank-order  changes  from  Session  1  to  Session  2;  the 
other  made  none.  The  pairwise  correlations  of  these  judges'  ranks  with  the  seven  other  judges' 
ranks,  shown  In  Table  2  for  judges  8  and  9,  are  at  the  lower  end  of  the  range  of  correlations. 
The  merits  of  the  shorter  procedure  may  be  worthy  of  further  Investigation,  however,  assuming 
only  an  overall  ranking  Is  uesired.  Present  results  are  Inconclusive. 

The  majo.'  unknown  factor  In  this  study  Is  the  validity  of  the  rankings.  The  divergent  judge 
may  have  been  right,  the  group  may  have  been  right,  or  both  may  have  been  wrong  about  the 
expected  payoff  of  the  projects.  To  assess  the  merit  of  the  rankings.  It  would  be  necessary  to 
Initiate  a  longitudinal  study,  ranking  the  projects  periodically,  and  keeping  a  record  of  those 
rankings  so  that  the  relationship  between  the  rankings  and  eventual  project  payoff  could  be 
determined. 


V.  CONCLUSIONS/RECOMMENDATIONS 

1.  Interjudge  rankings  were  sufficiently  reliable  to  suggest  that  rankings  based  on 
estimated  project  payoff  to  the  Air  Force  can  be  used  as  a  reliable  reference  In  distributing  R40 
resources. 

2.  The  feedback  session  allowed  judges  to  share  Information  about  project  merits  and  their 
proposed  ranks,  and  Increased  Interjudge  reliability. 

3.  For  Session  1,  Customer  Interest  was  the  factor  most  highly  correlated  with  the  overall 
rankings.  For  Session  2,  It  was  Payoff  to  the  Air  Force. 

4.  Project  Descriptions  should  depict  the  projects  In  terms  of  the  factors  on  which  the 
projects  are  being  evaluated.  Project  scientists  should  prepare  the  Project  Descriptions  In  a 
standard  format. 

5.  If  only  an  overall  ranking  Is  desired,  only  one  overall  ranking  may  be  necessary,  rather 
than  rankings  on  several  related  factors  and  then  on  an  overall  factor. 

6.  Different  ranking  factors  should  be  used  for  new  versus  old  projects.  Alternatively, 
small  amounts  of  seed  money  might  be  provided  for  several  new  projects,  rather  than  large  amounts 
of  money  for  a  few  highly  ranked  projects,  thereby  averaging  the  risks.  Projects  would  be  ranked 
after  being  underway  for  at  least  1  year. 

7.  Projects  should  be  ranked  twice  annually  for  at  least  5  years,  so  that  the  relationship 
between  ranks  and  eventual  project  pe^yoff  can  be  determined. 
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